Efficiently storing data in a cloud storage

ABSTRACT

A cloud server component determines that a size of a first cloud storage element object is at least below a first threshold. In response to the first determination, a client-side component is requested to store additional data in the cloud storage element object including by having the client-side component update the first cloud storage element with an updated version that includes previously existing data of the first cloud storage element and the additional data. The first cloud storage element object is added to a set of one or more cloud storage element objects available for update. The client-side component is configured to generate an updated version of the first cloud storage element object that has a size that is greater than or equal to the first threshold.

CROSS REFERENCE TO OTHER APPLICATIONS

This application is a continuation of U.S. Pat. Application No.17/402,206 entitled EFFICIENTLY STORING DATA IN A CLOUD STORAGE filedAug. 13, 2021, which claims priority to U.S. Provisional Pat.Application No. 63/074,682 entitled EFFICIENTLY STORING DATA IN A CLOUDSTORAGE filed Sep. 04, 2020, each of which is incorporated herein byreference for all purposes.

BACKGROUND OF THE INVENTION

Data Management as a Service (DMaaS) enables an entity (e.g.,enterprise, organization, government, company, user, individual, etc.)to manage data associated with the entity using one or more cloudservices provided by a cloud service provider. The cloud serviceprovider may charge the entity for its services based on the amount ofcloud storage used by the entity. The entity may copy its data from itsdatacenter to the cloud storage. However, merely copying data associatedwith the entity from the datacenter associated with the entity to thecloud storage provided by the cloud service provider may lead to aninefficient use of the cloud storage due to an increase in costsassociated with future operations on the copied data.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments of the invention are disclosed in the followingdetailed description and the accompanying drawings.

FIG. 1 is a block diagram illustrating a system for storing data in acloud storage in accordance with some embodiments.

FIG. 2 is a flow diagram illustrating a process for backing up data to acloud storage in accordance with some embodiments.

FIG. 3 is a flow diagram illustrating a process for determining whichdata chunks to back up to a cloud storage in accordance with someembodiments.

FIG. 4 is a flow diagram illustrating a process for writing data chunksto cloud storage in accordance with some embodiments.

FIG. 5 is a flow diagram illustrating a process for generating metadatain accordance with some embodiments.

FIG. 6 is a flow diagram illustrating a process for managing adeduplication table in accordance with some embodiments.

FIG. 7 is a flow diagram illustrating a process for updating a cloudstorage element object in accordance with some embodiments.

FIG. 8 is a flow diagram illustrating a process for updating a cloudstorage element object in accordance with some embodiments.

DETAILED DESCRIPTION

Techniques to efficiently store data in a cloud storage are disclosedherein. One technique to efficiently store data in the cloud storageincludes reducing the amount of data associated with an entity stored ina cloud storage by deduplicating the data associated with the entitystored in the cloud storage. The cloud storage provider may providecloud storage for a plurality of entities. Each of the entities may bereferred herein to as a “storage tenant.”

A data plane may be configured to perform one or more data managementservices (e.g., backup, tiering, replication, migration, etc.) andcontrol how data associated with a storage tenant is stored in the cloudstorage. For example, the data plane may include a backup engine that isconfigured to determine when a source system associated with the storagetenant is to perform a backup, a tiering engine that is configured todetermine when an object is to be tiered from the source system to cloudstorage, and/or a replication engine that is configured to determinewhen data content is to be replicated from the source system to cloudstorage. The source system may be part of a datacenter that includes aclient-side component. The client-side component may be a virtualmachine, a container, a server, an application, etc. The client-sidecomponent may be configured to establish and securely communicate withthe data plane via a bidirectional network connection (e.g., gRPC (gRPCremote procedure call) connection).

The client-side component may receive from the backup engine aspecification of content to be backed up from the source system to thecloud storage. In some embodiments, the specification of contentdescribes a full backup of the source system. In some embodiments, thespecification of content describes an incremental backup of the sourcesystem. In some embodiments, the specification of content describes afull backup of an object included in the source system (e.g., virtualmachine, container, application, storage cluster). In some embodiments,the specification of content describes an incremental backup of anobject included in the source system. In some embodiments, thespecification of content describes a storage tier for one or moreobjects stored on the source system (e.g., one or more files, one ormore virtual machines, one or more containers, one or more applications,one or more databases, etc.). For example, a backup of a file stored onthe source system may be tiered to a first performance storage classassociated with a cloud storage, a second performance storage classassociated with the cloud storage, or a third performance storage classassociated with the cloud storage.

The client-side component may receive from a tiering engine aspecification of content to be tiered from a source system to the cloudstorage. In some embodiments, the client-side component determines aspecification of content to be tiered from the source system to thecloud storage. In some embodiments, the specification of contentdescribes a storage tier for one or more objects stored on the sourcesystem. For example, a file stored on the source system may be tieredfrom the source system to a first performance storage class associatedwith a cloud storage, a second performance storage class associated withthe cloud storage, or a third performance storage class associated withthe cloud storage.

The client-side component may receive from a replication engine aspecification of content to be replicated from source system to thecloud storage. In some embodiments, the specification of contentdescribes a storage tier for one or more objects stored on the sourcesystem. For example, a file stored on the source system may bereplicated from the source system to a first performance storage classassociated with a cloud storage, a second performance storage classassociated with the cloud storage, or a third performance storage classassociated with the cloud storage.

The client-side component may be configured to request and receive thecontent in a plurality of portions from the source system. A receivedportion may include metadata associated one or more objects stored onthe source system (e.g., owner, created, last update, size, permissions,etc.) and/or data content associated with the one or more objects storedon the source system. For each received portion of content, theclient-side component divides the received portion of content into aplurality of data chunks. In some embodiments, a specification ofcontent to be stored in the cloud storage is received from a backupengine. In some embodiments, a specification of content to be stored inthe cloud storage is received from a source system. In some embodiments,the plurality of data chunks are of variable size. The client-sidecomponent may generate corresponding chunk identifiers (e.g., securehash algorithm 1 (SHA-1) identifier) for each of the plurality of datachunks included in a portion of received content. Data chunk metadatathat includes the corresponding chunk identifiers is sent to a filesystem manager that is included in a cloud server of the data plane. Foreach data chunk, the data chunk metadata may also include acorresponding data chunk size and a corresponding object offset. Thedata chunk metadata indicates a sequence of data chunks within anobject. For example, the data chunk metadata may indicate that a firstdata chunk having a chunk identifier of C1 is associated with an objectoffset of 0 MB - 1 MB and has a data chunk size of 1 MB, a second datachunk having a chunk identifier of C2 is associated with an objectoffset of 1 MB - 2 MB and has a data chunk of 1 MB, ..., and an 8th datachunk identifier of C8 is associated with an object offset of 7 MB - 8MB and has a data chunk size of 1 MB.

In response to receiving the corresponding chunk identifiers for each ofthe plurality of data chunks, the file system manager is configured tocompare each of the corresponding chunk identifiers to chunk identifiersincluded in a deduplication table stored at the data plane. The chunkidentifiers included in the deduplication table indicate at least someof the data chunks associated with the storage tenant that are alreadystored in the cloud storage. If the client-side component were to writeto the cloud storage data chunks that are already stored at the cloudstorage, then the cloud storage would store duplicate copies of the samedata chunks, which is an inefficient use of the cloud storage.

The file system manager is configured to provide a data structure to theclient-side component. In some embodiments, the data structure indicatesone or more data chunks requested by the file system manager (e.g., theone or more chunk identifiers of the one or more received chunkidentifiers that do not correspond to data chunks stored at the cloudstorage). In some embodiments, the data structure indicates the one ormore chunk identifiers that correspond to one or more data chunks thatare stored at the cloud storage, but the one or more chunk identifiersassociated with the one or more data chunks that are stored at the cloudstorage are not included in deduplication table due to one or morefactors, such as whether the data chunk is referenced by a tree datastructure that was generated after a particular date, a recency of whenthe data chunk was last deduplicated, a frequency at which the datachunk is deduplicated, a service level agreement, a storage tier of adata chunk, etc. In response to receiving the data structure, theclient-side component is configured to use the data structure toidentify the one or more data chunks of the portion of the receivedcontent that are to be sent to be stored at the cloud storage.

The file system manager may also send to the client-side component areference to one or more portions of one or more cloud storage elementobjects of the cloud storage to which the client-side component maywrite the one or more data chunks associated with the one or more chunkidentifiers included in the data structure. The reference may includecorresponding identifiers for the one or more cloud storage elementobjects. In some embodiments, the one or more cloud storage elementobjects include one or more new cloud storage element objects, one ormore existing cloud storage element objects, and/or one or more cloudstorage element objects yet to be created. In some embodiments, thereference may comprise one or more identifiers which the client-sidecomponent may use to create new cloud storage element objects of thecloud storage.

The client-side component may receive an encryption key from a cloudserver that is located in the data plane. The cloud server may includean encryption key manager that manages encryption keys for a pluralityof storage tenants. The received encryption key is particular to thestorage tenant. In some embodiments, the encryption key enables readaccess for a cloud storage element object associated with the storagetenant at a credential level or write access for a new cloud storageelement object associated with the storage tenant at a credential levelto prevent overwriting or deleting of existing files. That is, writeaccess is not permitted for existing files. For example, some cloudstorage element objects may each have a unique credential. In someembodiments, the encryption key enables read or write access for a cloudstorage element object associated with the storage tenant at a prefixlevel. For example, a prefix may be a partial or full file path in thecloud storage. A prefix may be unique to a storage tenant. Some or allcloud storage element objects of a particular storage tenant may bestored in cloud storage and share a prefix. The encryption key mayexpire after a particular amount of time. A source system associatedwith the storage tenant may be hacked and having the encryption keyexpire after the particular amount of time may prevent a hacker fromaccessing any of the cloud storage element objects associated with thestorage tenant that are stored at the cloud storage.

Another technique to efficiently store data in the cloud storageincludes combining a plurality of the identified data chunks into one ormore batches. In some embodiments, a default size of a cloud storageelement object is specified (e.g., 8 MB). In some embodiments, a size ofa batch of data chunks is the default size of the cloud storage elementobject. Storing this particular batch of data chunks at the cloudstorage is an efficient use of the cloud storage. In some embodiments, asize of a batch of data chunks is less than the default size of thecloud storage element object, but greater than or equal to a thresholdsize. This is a sub-optimal configuration of the cloud storage due tooverhead costs associated with future operations on the cloud storageelement objects. For example, storing a large number of cloud storageelement objects associated with an object that are less than the defaultsize increases the number of reads needed to restore the object, whichincreases the overall amount of time needed to restore the object. Insome embodiments, a size of a batch of data chunks is less than thedefault size of the cloud storage element object and less than thethreshold size. Storing batches of data chunks that are less than thedefault size of the cloud storage element object and less than thethreshold size may also increase transaction and/or storage costsassociated with the batch of data chunks. For example, storing a largenumber of cloud storage element objects associated with an object thatare less than the threshold size also increases the number of readsneeded to restore the object, which increases the overall amount of timeneeded to restore the object.

In some embodiments, a default range size of a cloud storage elementobject is specified (e.g., 4 MB - 12 MB). In some embodiments, the sizeof a batch of data chunks is within the default size range of a cloudstorage element object. In some embodiments, the size of a batch of datachunks is not within the default size range of a cloud storage elementobject.

Prior to writing a batch of data chunks to a cloud storage elementobject that was indicated by the file system manager of the data plane,the client-side component may compare a batch size of a batch to athreshold size (e.g., 4 MB) or a default size range (e.g., 4 MB - 12MB). Additional overhead costs may be incurred by storing at the cloudstorage cloud storage element objects that have a size that is less thanthe threshold size or outside the default size range. Such cloud storageelement objects may require additional application programming interface(API) calls to access the cloud storage element objects, which may addto the overall time and resources needed to perform a data managementfunction (e.g., garbage collection, deduplication, replication, backup,etc.).

In the event the batch size of a batch is greater than or equal to thethreshold size, the client-side component may encrypt the data chunksincluded in the batch with the received encryption key and write theencrypted data chunks to one of the cloud storage element objectsidentified by the file system manager. In some embodiments, the datachunks are compressed prior to being encrypted. An index may be updatedto indicate which data chunks are included in the cloud storage elementobject. In some cases, the index may indicate the relative position ofdata chunks in the cloud storage element object. The index may includean object id and a version number. The index may be included in thecloud storage element object. In some embodiments, the client-sidecomponent generates a cloud storage element object identified by thefile system manager. In some embodiments, a cloud storage generates thecloud storage element object identified by the file system manager.

In the event the batch size of the batch is not greater than or equal tothe threshold size, the client-side component may determine whether abatch period greater than or equal to a batch threshold period haspassed. The client-side component may store in-memory the data chunk(s)included in the batch for the batch threshold period (e.g., <1 seconds)to allow additional data chunks to be included in the batch.

In the event the batch period is not greater than or equal to the batchthreshold period, the client-side component processes a subsequentportion of the specified content to identify one or more data chunks ofthe subsequent portion of the specified content that are to be stored inthe cloud storage. The one or more data chunks of the subsequent portionare combined with the one or more data chunks of the previous portion togenerate a new batch of data chunks. The client-side componentdetermines whether a size of the new batch of data chunks is greaterthan the threshold size. If it is, then the client-side componentdetermines whether the batch period is greater than or equal to thebatch threshold period. If it is not, then the above process repeatsuntil the batch threshold period is greater than or equal to the batchthreshold period.

In the event the batch period is greater than or equal to the batchthreshold period, the client-side component may write the one or moredata chunks included in the batch to a storage of a cloud serverincluded in the data plane, even if the size of the batch written to thestorage is less than the threshold size. In response to receiving theone or more data chunks, the file system manager of the cloud server mayaggregate the one or more received data chunks with one or more otherreceived data chunks. The file system manager may determine whether acumulative size of the aggregated data chunks is greater than thethreshold size and less than a maximum size for a cloud storage elementobject. In the event the cumulative size of the aggregated data chunksis greater than the threshold size and less than the maximum size forthe cloud storage object, the file system manager may batch theaggregated data chunks into a cloud storage element object, encrypt thebatched data chunks using an encryption key particular to the storagetenant, and write the batched data chunks to a cloud storage elementobject associated with the cloud storage. This is an efficient use ofthe cloud storage because instead of storing the one or more receiveddata chunks in a first cloud storage element object and storing the oneor more other received data chunks in a second cloud storage elementobject, the one or more received data chunks and the one or more otherreceived data chunks are stored in a single cloud storage elementobject.

In some embodiments, the cloud storage element object associated withthe cloud storage is a new cloud storage element object. In someembodiments, the cloud storage element object associated with the cloudstorage is a cloud storage element object that is yet to be created. Insome embodiments, the cloud storage element object associated with thecloud storage is an existing cloud storage element object. In the eventthe cloud storage element object associated with the cloud storage is anexisting cloud storage element object, the file system manager may readthe data chunks associated with the existing cloud storage elementobject, combine the read data chunks with one or more new data chunks togenerate a new version of the cloud storage element object, create anindex for the new version of the cloud storage element object, andinclude the index in the new version of the cloud storage elementobject. In some embodiments, multiple threads write to the same cloudstorage element object. In some cases, the index may indicate therelative position of data chunks in the cloud storage element object.The index may include an object id and a version number.

In the event the cumulative size of the aggregated data chunks is notgreater than the threshold size, the file system manager may determineif the total time that the one or more received data chunks have beenstored in a storage of a cloud server included in the data plane isgreater than a threshold time. In the event the total time is greaterthan the threshold time, then the file system manager may batch the oneor more received data chunks and one or more other received data chunks,if any, encrypt the batched data chunks using an encryption keyparticular to the storage tenant, and write the batched data chunks to acloud storage element object associated with a cloud storage. This maycause overhead costs associated with future operations on the cloudstorage element objects to be expensive because operations on smallcloud storage element objects are inefficient, but may be cost effectiveand improve resiliency as the storage of a cloud server (e.g., EC2instance) may be less reliable and/or more expensive than the cloudstorage (e.g., S3). The one or more data chunks may be written to acloud storage element object stored at the cloud storage to ensure thatthe one or more data chunks are not lost in the event the storage of thecloud server fails. In some embodiments, the cloud storage elementobject associated with the cloud storage is a new cloud storage elementobject. In some embodiments, the cloud storage element object associatedwith the cloud storage is a cloud storage element object yet to becreated. In some embodiments, the cloud storage element objectassociated with the cloud storage is an existing cloud storage elementobject. In the event the cloud storage element object associated withthe cloud storage is an existing cloud storage element object, the filesystem manager may read the data chunks associated with the existingcloud storage element object, combine the read data chunks with one ormore new data chunks to generate a new version of the cloud storageelement object, create an index for the new version of the cloud storageelement object, and include the index in the new version of the cloudstorage element object.

Another technique to efficiently store data in the cloud storageincludes monitoring the cloud storage to identify cloud storage objectsthat are less than a default size or outside of the default range size.A file system manager may scan the cloud storage to identify a cloudstorage element object. A size of the cloud storage element object isdetermined. The determined size is compared to a first threshold size(e.g., 1 MB). In the event the determined size is less than the firstthreshold size, the file system manager may add the cloud storageelement object to a set of one or more cloud storage element objectsavailable for a client-side component to update the cloud storageelement with an updated version. The updated version may includepreviously existing data of the cloud storage element object andadditional data. In some embodiments, a size of the updated version ofthe cloud storage element object is the default size for a cloud storageelement object. In the event the determined size is not less than thefirst threshold size, but less than a second threshold size, the filesystem manager updates the cloud storage element object with an updatedversion that includes previously existing data of the cloud storageelement object and the new data chunks received from the client-sidecomponent.

FIG. 1 is a block diagram illustrating of a system for storing data in acloud storage in accordance with some embodiments. In the example shown,system 100 includes a client-side component 104 that is connected todata plane 112 via connection 110 and connected to cloud storage 122 viaconnection 120. In some embodiments, connection 110 is a gRPC connectionor other type of connection. In some embodiments, connection 120 is aHTTPS connection or other type of connection. In some embodiments,connections 110, 120 may be intermittent connections. The connectionbetween client-side component 104 and data plane 112 or betweenclient-side component 104 and cloud storage 122 may be a wireless orwired connection. In some embodiments, the connection betweenclient-side component 104 and data plane 112 or between client-sidecomponent 104 and cloud storage 122 may be a gRPC connection.Connections 110, 120 may communicate data and/or information via a localarea network, a wide area network, a storage area network, campus areanetwork, metropolitan area network, system area network, intranet, theInternet, and/or a combination thereof.

One or more client-side components 104 are located in datacenter 102(e.g., real datacenter or virtual datacenter) that is associated with astorage tenant. In some embodiments, one or more client-side components104 are located in cloud environment 121, such as in data plane 112. Aclient-side component may be a virtual machine, a container, a server,an application, etc. The one or more client-side components 104 areconnected (e.g., temporarily or persistently) to source system 103. Insome embodiments, the one or more client-side components 104 areincluded in source system 103. In some embodiments, the one or moreclient-side components are included in a storage system 105. In someembodiments, source system 103 is a primary system. In some embodiments,source system 103 is a secondary storage system that backed up data fromother sources (not shown). A primary system may be comprised of one ormore computing devices (e.g., servers, desktops, laptops, etc.). Storagesystem 105 may be comprised of one or more storage nodes that includeone or more corresponding processors, one or more correspondingmemories, and one or more corresponding storage devices.

In some embodiments, a storage node of storage system 105 includes aprocessor, memory, and a plurality of storage devices. The plurality ofstorage devices may include one or more solid state drives, one or morehard disk drives, or a combination thereof. Backed up data may be storedin the one or more solid state drives, one or more hard disk drives, ora combination thereof.

In some embodiments, a storage node of storage system 105 includes aprocessor and memory, and is coupled to a separate storage device. Theseparate storage device may include one or more storage devices (e.g.,flash storage devices). A storage device may be segmented into aplurality of partitions. Each of the storage nodes may be allocated oneor more of the partitions. The one or more partitions allocated to astorage node may be configured to store data associated with some or allof the changed objects that were backed up to the storage system. Forexample, the separate storage device may be segmented into 10 partitionsand the storage system may include 10 storage nodes. A storage node ofthe 10 storage nodes may be allocated one of the 10 partitions.

In some embodiments, a storage node of storage system 105 includes aprocessor, memory, and a storage device. A storage device may besegmented into a plurality of partitions. Each of the storage nodes maybe allocated one or more of the partitions. The one or more partitionsallocated to a storage node may be configured to store data associatedwith some or all of the changed objects that were backed up to thestorage system. For example, the separate storage device may besegmented into 10 partitions and the storage system may include 10storage nodes. A storage node of the 10 storage nodes may be allocatedone of the 10 partitions.

In some embodiments, the storage nodes of the storage system arehomogenous nodes where each storage node has the same capabilities(e.g., processing, storage, memory, etc.). In some embodiments, at leastone of the storage nodes of the storage system is a heterogeneous nodewith different capabilities (e.g., processing, storage, memory, etc.)than the other storage nodes of storage system.

Source system 103 includes content (e.g., virtual machines,applications, files, filesystem data, containers, etc.) that is to bebacked up to cloud storage 122. A client-side component is configured toreceive the content to be backed up from source system 103 and to backup the received content. In some embodiments, the client-side componentis configured to back up some or all of the received content to cloudstorage 122. In some embodiments, the client-side component isconfigured to back up some of the received content to storage 116included in data plane 112.

Data plane 112 is configured to orchestrate how data associated with astorage tenant is stored in cloud storage 122. For example, dataassociated with a storage tenant may be stored in a first performancestorage class 123 (e.g., Amazon Simple Storage Service (S3)), a secondperformance storage class 124 (e.g., Amazon S3 Glacier), or a thirdperformance storage class 125 (e.g., Amazon S3 Glacier Deep Archive).Although FIG. 1 depicts cloud storage 122 as having three performancetiers, cloud storage 122 may have n performance tier. Each of theperformance tiers corresponds to an object storage provided by a cloudprovider (e.g., Amazon Web Services™, Microsoft™ Azure, Google Cloud™,etc.) that provides cloud environment 121 (e.g., public cloud, privatecloud, hybrid cloud, etc.). The speed at which data may be accessed isdifferent depending on whether the data is associated with the firstperformance storage class 123, the second performance storage class 124,or the third performance storage class 125.

Data plane 112 includes one or more cloud servers 113 (e.g., one or moreEC2 instances). The one or more cloud servers 113 may be associated withcorresponding processors, corresponding memory, and correspondingstorage. In some embodiments, data plane 112 includes multiple sets ofone or more cloud servers 113. For example, data plane 113 may include afirst set of one or more cloud servers 113 that is associated with afirst region and a second set of one or more cloud servers 113 that isassociated with a second region. The one or more cloud servers 113include a deduplication table 114 that associates chunk identifiers withdata chunks stored in cloud storage 122. In some embodiments,deduplication table 114 associates chunk identifiers with data chunksassociated with the first performance storage class 123, but not datachunks associated with the second performance storage class 124 or thethird performance storage class 125. This may reduce the overall size ofdeduplication table 114 maintained by file system manager 115. In someembodiments, deduplication table 114 stores chunk identifiers for aportion of the data chunks that are associated with the firstperformance storage class 123. For example, the chunk identifiersassociated with the most popular data chunks (e.g., data chunksreferenced by a threshold number of files or objects) or the most recentdata chunks (e.g., data chunks that were stored to cloud storage 122within a particular period of time) may be included in deduplicationtable 114.

Data plane 112 includes backup engine 118 that is configured todetermine when source system 103 is to perform a backup according to abackup policy. The backup policy may indicate a frequency at which abackup is to be performed (e.g., hourly, daily, weekly, etc.), an amountof change data for which a backup is to be performed (e.g., more than 10GB of changed data), or in response to a user command. Backup engine 118may provide to the one or more client-side components 104 aspecification of content to be backed up from source system 103 to cloudstorage 122. In some embodiments, the specification of content describesa full backup of source system 103. In some embodiments, thespecification of content describes an incremental backup of sourcesystem 103. In some embodiments, the specification of content describesa full backup of an object included in source system 103 (e.g., virtualmachine, container, application). In some embodiments, the specificationof content describes an incremental backup of an object included insource system 103. In some embodiments, the specification of contentdescribes a storage tier for one or more objects stored on source system103. For example, a backup of a file stored on source system 103 may betiered to the first performance storage class 123, the secondperformance storage class 124, or the third performance storage class125.

Data plane 112 includes tiering engine 111. A client-side component mayreceive from tiering engine 111 a specification of content to be tieredfrom source system 103 to cloud storage 122. In some embodiments, thespecification of content describes a storage tier for one or moreobjects stored on source system 103. For example, a file stored onsource system 103 may be tiered from source system 103 to a firstperformance storage class 123, a second performance storage class 124,or a third performance storage class 125.

Data plane 112 includes replication engine 131. A client-side componentmay receive from replication engine 131 a specification of content to bereplicated from source system 103 to cloud storage 122. In someembodiments, the specification of content describes a storage tier forone or more objects stored on source system 103. For example, a filestored on source system 103 may be replicated from source system 103 toa first performance storage class 123, a second performance storageclass 124, or a third performance storage class 125.

The one or more client-side components 104 are configured to receive thecontent in a plurality of portions from source system 103. For eachreceived portion of content, a client-side component divides thereceived portion of content into a plurality of data chunks. In someembodiments, a specification of content to be stored in cloud storage122 is received from backup engine 118. In some embodiments, aspecification of content to be stored in cloud storage 122 is receivedfrom source system 103. In some embodiments, the plurality of datachunks may be of variable size. The one or more client-side components104 may generate corresponding chunk identifiers (e.g., SHA-1identifier, SHA-2 identifier, SHA-256 identifier, etc.) for each of theplurality of data chunks included in a portion of received content.

A client-side component may identify one or more data chunks of theplurality of data chunks that are to be sent via network connection 120to cloud storage 122 at least in part by sending to file system manager115 the corresponding chunk identifiers for each of the plurality ofdata chunks included in a portion of received content. In response toreceiving the corresponding chunk identifiers for each of the pluralityof data chunks, file system manager 115 is configured to compare each ofthe corresponding chunk identifiers to chunk identifiers included indeduplication table 114. The chunk identifiers included in deduplicationtable 114 indicate at least some of the data chunks that are stored incloud storage 122.

File system manager 115 is configured to provide the client-sidecomponent a data structure that indicates the one or more chunkidentifiers of the one or more received chunk identifiers that are notincluded in deduplication table 114. In some embodiments, the one ormore chunk identifiers not included in deduplication table 114correspond to data chunks that are not stored at cloud storage 122. Insome embodiments, the one or more chunk identifiers not included in thededuplication table correspond to one or more data chunks that arestored at cloud storage 122, but the one or more chunk identifiersassociated with the one or more data chunks that are stored at cloudstorage 122 are not included in deduplication table 114 due to one ormore factors, such as whether the data chunk is referenced by a treedata structure that was generated after a particular date, a recency ofwhen the data chunk was last deduplicated, a frequency at which the datachunk is deduplicated, a service level agreement, a storage tier of adata chunk, etc.. In response to receiving the data structure, theclient-side component is configured to use the data structure toidentify the one or more data chunks of the portion of the receivedcontent that are to be sent via network connection 120 to be stored atcloud storage 122.

File system manager 115 may also provide an indication of one or morecloud storage element objects of cloud storage 122 to which aclient-side component may write the one or more data chunks associatedwith the one or more chunk identifiers included in the data structure.In some embodiments, the indication includes a size of a batch of datachunks to create. In some embodiments, the indication of the one or morecloud storage element objects include corresponding cloud storageelement object identifiers. In some embodiments, the one or more cloudstorage element objects include one or more new cloud storage elementobjects, one or more existing cloud storage element objects, and/or oneor more cloud storage element objects yet to be created. In someembodiments, file system manager 115 periodically (e.g., every hour)provides to the one or more client-side components 104 an indication ofa set of one or more cloud storage element objects.

A client-side component may receive an encryption key from encryptionkey manager 119. Encryption key manager manages encryption keys for aplurality of storage tenants. The received encryption key is particularto the storage tenant. In some embodiments, the encryption key enablesread for a cloud storage element object associated with the storagetenant at a credential level or write access for a new cloud storageelement object associated with the storage tenant at a credential level.For example, some cloud storage element objects may each have a uniquecredential. In some embodiments, the encryption key enables read orwrite access for a cloud storage element object associated with thestorage tenant at a prefix level. For example, a prefix may be a partialor full file path in the cloud storage. A prefix may be unique to astorage tenant. Some or all cloud storage element objects of aparticular storage tenant may be stored in cloud storage and share aprefix. The encryption key may expire after a particular amount of time.Source system 103 may be hacked and having the encryption key expireafter the particular amount of time may prevent a hacker from accessingany of the cloud storage element objects stored at cloud storage 122.

A client-side component may combine a plurality of the identified datachunks into one or more batches. A default size of a cloud storageelement object may be specified. In some embodiments the default size ofthe cloud storage object is 8 MB. In some embodiments, a size of thecloud storage element object is 8 MB. Storing a batch of default size atcloud storage 122 is an efficient use of the cloud storage. In someembodiments, a size of a batch of data chunks is less than the defaultsize of the cloud storage element object, but greater than or equal to athreshold size. This is a sub-optimal configuration of cloud storage 122due to overhead costs associated with future operations on the cloudstorage element objects. In some embodiments, a size of a batch of datachunks is less than the default size of the cloud storage element objectand less than the threshold size.

In some embodiments, a default range size of a cloud storage elementobject is specified (e.g., 4 MB - 12 MB). In some embodiments, the sizeof a batch of data chunks is within the default size range of a cloudstorage element object. In some embodiments, the size of a batch of datachunks is not within the default size range of a cloud storage elementobject.

Prior to writing a batch of data chunks to a cloud storage elementobject of cloud storage 122, a client-side component may compare a batchsize of a batch to a threshold size (e.g., 4 MB). Additional overheadcosts may be incurred by storing at cloud storage 122 cloud storageelement objects that have a size that is less than the threshold size.Such cloud storage element objects may require additional applicationprogramming interface (API) calls to access the cloud storage elementobjects, which may add to the overall time and resources needed toperform a data management function (e.g., garbage collection,deduplication, replication, backup, etc.).

In the event the batch size of a batch is greater than or equal to thethreshold size, the client-side component may encrypt the data chunksincluded in the batch with the received encryption key and write theencrypted data chunks to one of the cloud storage element objectsidentified by file system manager 115. In some embodiments, the datachunks are compressed prior to being encrypted. The client-sidecomponent may write the encrypted data chunks to a cloud storage elementobject associated with the first performance storage class 123, thesecond performance storage class 124, or the third performance storageclass 125. The performance storage class to which the client-sidecomponent 104 writes the batch of data chunks may depend on a servicelevel agreement associated with a storage tenant. In the event the batchsize of the batch is not greater than or equal to the threshold size,the one or more client-side components 104 write one or more data chunksincluded in the batch to storage 116 of the one or more cloud servers113. In some embodiments, the cloud storage element object associatedwith the cloud storage is a new cloud storage element object. In someembodiments, the cloud storage element object associated with the cloudstorage is an existing cloud storage element object. In the event thecloud storage element object associated with the cloud storage is anexisting cloud storage element object, the file system manager may readthe data chunks associated with the existing cloud storage elementobject, combine the read data chunks with one or more new data chunks togenerate a new version of the cloud storage element object, create anindex for the new version of the cloud storage element object, andinclude the index in the new version of the cloud storage elementobject. In some embodiments, the index includes file offsets associatedwith the data chunks included in the cloud storage element object. Insome embodiments, the index includes a pointer to a group of data chunksthat were compressed and included in the cloud storage element object.The index may include an object id and a version number.

In the event the batch size of a batch is not greater than or equal tothe threshold size, the client-side component may determine whether abatch period greater than or equal to a batch threshold period haspassed. The client-side component may store in-memory the data chunk(s)included in the batch for the batch threshold period (e.g., <1 seconds)to allow additional data chunks to be included in the batch.

In the event the batch period is not greater than or equal to the batchthreshold period, the client-side component processes a subsequentportion of the specified content to identify one or more data chunks ofthe subsequent portion of the specified content that are to be stored inthe cloud storage. The one or more data chunks of the subsequent portionare combined with the one or more data chunks of the previous portion togenerate a new batch of data chunks. The client-side componentdetermines whether a size of the new batch of data chunks is greaterthan the threshold size. If it is, then the client-side componentdetermines whether the batch period is greater than or equal to thebatch threshold period. If it is not, then the above process repeatsuntil the batch threshold period is greater than or equal to the batchthreshold period.

In the event the batch period is greater than or equal to the batchthreshold, the client-side component may provide one or more data chunksincluded in the batch to cloud server 113. In response to receiving theone or more data chunks, file system manager 115 may aggregate the oneor more received data chunks with one or more other received datachunks. In some embodiments, the one or more received data chunks andthe one or more other received data chunks are received from the sameclient-side component. In some embodiments, the one or more receiveddata chunks and the one or more other received data chunks are receivedfrom a plurality of client-side components. In some embodiments, filesystem manager 115 determines whether a cumulative size of theaggregated data chunks is greater than the threshold size and less thana maximum size for a cloud storage object. In some embodiments, filesystem manger 115 determines whether a cumulative size of the aggregateddata chunks is within a default size range. In the event the cumulativesize of the aggregated data chunks is greater than the threshold sizeand less than the maximum size for the cloud storage object, or withinthe default size range, file system manager 115 may batch the aggregateddata chunks, encrypt the batched data chunks using an encryption keyassociated with the storage tenant, and write the batched data chunks toa cloud storage element object associated with cloud storage 122 viaconnection 130. In some embodiments, connection 130 is a HTTPSconnection or other type of connection. In some embodiments, the datachunks are compressed prior to being encrypted. File system manager 115may write the encrypted data chunks to a cloud storage element objectassociated with the first performance storage class 123, the secondperformance storage class 124, or the third performance storage class125. In the event the cumulative size of the aggregated data chunks isnot greater than the threshold size, file system manager 115 maydetermine if the total time that the one or more received data chunkshave been stored in storage 116 is greater than a threshold time. In theevent the total time is greater than the threshold time, then filesystem manager 115 may batch the one or more received data chunks andone or more other received data chunks, if any, encrypt the batched datachunks using an encryption key associated with the storage tenant, andwrite the batched data chunks to a cloud storage element objectassociated with cloud storage 122.

In some embodiments, the cloud storage element object associated withthe cloud storage is a new cloud storage element object. In someembodiments, the cloud storage element object associated with the cloudstorage is a cloud storage element object yet to be created. In someembodiments, the cloud storage element object associated with the cloudstorage is an existing cloud storage element object. In the event thecloud storage element object associated with the cloud storage is anexisting cloud storage element object, the file system manager may readthe data chunks associated with the existing cloud storage elementobject, combine the read data chunks with one or more new data chunks togenerate a new version of the cloud storage element object, create anindex for the new version of the cloud storage element object, andinclude the index in the new version of the cloud storage elementobject. The index may include an object id and a version number. In theevent the total time is not greater than the threshold time, then filesystem manager 115 continues to aggregate a plurality of data chunks.

After the one or more client-side components 104 have written some orall of one or more batches of data chunks to one or more cloud storageelement objects of cloud storage 122, the one or more client-sidecomponents 104 may be configured to provide file system manager 115 anindication that the one or more cloud storage element objects have beenfinalized (e.g., the one or more data chunks have been stored at areferenced portion of cloud storage 122). A cloud storage element objectmay be finalized when cloud storage 122 receives and stores all of theplurality of data chunks associated with the cloud storage elementobject. In response to receiving the indication, file system manager 115is configured to generate metadata for the one or more data chunksstored in cloud storage 122 by one of the one or more client-sidecomponents 104. The metadata for the one or more data chunks may includea tree data structure that organizes the one or more data chunks. Anexample of the tree data structure is a snapshot tree, which may bebased on a B+ tree structure (or other type of tree structure in otherembodiments). An example of a tree data structure is described in U.S.Pat. Application 16/287,214 entitled “Deploying A Cloud Instance Of AUser Virtual Machine,” filed on Feb. 27, 2019, which is incorporatedherein by reference for all purposes.

FIG. 2 is a flow diagram illustrating a process for backing up data to acloud storage in accordance with some embodiments. In the example shown,process 200 may be implemented by a client-side component, such as oneof the one or more client-side components 104.

At 202, a specification of content to be stored in a cloud storage isreceived. The specification may be received from a backup engine. Insome embodiments, the backup engine is part of a data plane that islocated in a cloud environment provided by a cloud service provider. Insome embodiments, the specification is received from a source system.

In some embodiments, the specification of content describes a fullbackup of a source system. In some embodiments, the specification ofcontent describes an incremental backup of a source system. In someembodiments, the specification of content describes a plurality ofbackups of a source system. In some embodiments, the specification ofcontent describes a full backup of an object included in the sourcesystem (e.g., virtual machine, container, database, application). Insome embodiments, the specification of content describes an incrementalbackup of an object included in a source system. In some embodiments,the specification of content describes a storage tier for one or moreobjects stored on a source system. For example, a backup of a filestored on source system may be tiered to the first performance storageclass, the second performance storage class, or the third performancestorage class.

At 204, a portion of the specified content is received at a client-sidecomponent from a source system. For example, 8 MB of a 1 TB backup maybe received. Multiple portions of the specified content are received atthe client-side component from the source system until a data managementoperation (e.g., backup, tiering, replication) is completed. The portionof the specified content may include metadata associated one or moreobjects stored on the source system (e.g., owner, created, last update,size, permissions, etc.) and/or data content associated with the one ormore objects stored on the source system.

At 206, the received portion of the specified content is divided into aplurality of data chunks. In some embodiments, the plurality of datachunks may be of variable size.

At 208, one or more data chunks of the plurality of data chunks to besent via a network to be stored in the cloud storage are identified. Acorresponding chunk identifier is computed for each of the one or moredata chunks. Data chunk metadata that includes the one or morecorresponding chunk identifiers, corresponding data chunk size, andcorresponding object offsets is sent to a file system manager. The datachunk metadata indicates a sequence of data chunks within an object. Forexample, the data chunk metadata may indicate that a first data chunkhaving a chunk identifier of C1 is associated with an object offset of 0MB - 1 MB and has a data chunk size of 1 MB, a second data chunk havinga chunk identifier of C2 is associated with an object offset of 1 MB - 2MB and has a data chunk of 1 MB, ..., and an 8th data chunk identifierof C8 is associated with an object offset of 7 MB - 8 MB and has a datachunk size of 1 MB. In response to receiving the one or morecorresponding chunk identifiers, the file system manager may determinewhether any of the one or more corresponding chunk identifiers areincluded in a deduplication table by comparing each of the correspondingchunk identifiers to chunk identifiers included in a deduplicationtable. In some embodiments, the one or more chunk identifiers notincluded in the deduplication table correspond to data chunks that arenot stored at a cloud storage. In some embodiments, the one or morechunk identifiers not included in the deduplication table correspond toone or more data chunks that are stored at the cloud storage; however,data chunks which are stored at the cloud storage may not be included inthe deduplication table due to one or more factors, such as whether thedata chunk is referenced by a tree data structure that was generatedafter a particular date, a recency of when the data chunk was lastdeduplicated, a frequency at which the data chunk is deduplicated, aservice level agreement, a storage tier of a data chunk, etc. The filesystem manager may identify which of the one or more corresponding chunkidentifiers correspond to data chunks that are not included in thededuplication table stored at the cloud storage and provide to aclient-side component a data structure that includes the identified oneor more corresponding chunk identifiers.

At 210, an encryption key and a reference to a portion of a cloudstorage where one or more data chunks are to be stored are received froma cloud server. The encryption key is particular to a storage tenant.The encryption key may expire after a particular amount of time. In someembodiments, the encryption key enables read access for a cloud storageelement object associated with the storage tenant at a credential levelor write access for a new cloud storage element object associated withthe storage tenant at a credential level. For example, some cloudstorage element objects may each have a unique credential. In someembodiments, the encryption key enables read or write access for a cloudstorage element object associated with the storage tenant at a prefixlevel. For example, a prefix may be a partial or full file path in thecloud storage. A prefix may be unique to a storage tenant. Some or allcloud storage element objects of a particular storage tenant may bestored in cloud storage and share a prefix.

The reference to a portion of the cloud storage may identify one or morecloud storage element objects of the cloud storage to which one or moredata chunks are to be stored. For example, the reference to the portionof the cloud storage may include corresponding identifiers for a set ofone or more cloud storage element objects. In some embodiments, the setof one or more cloud storage element objects include one or more newcloud storage element objects, one or more existing cloud storageelement objects, and/or one or more cloud storage element objects yet tobe created. In some embodiments, a reference may comprise a cloudstorage element object name which is to be used to create a new cloudstorage element object. In some embodiments, the reference to one ormore cloud storage element objects is periodically received from thedata plane. In some embodiments, the one or more cloud storage elementobjects included in the reference are available to be written for athreshold period of time (e.g., 1 hour).

The one or more identified cloud storage element objects may be storedin first performance storage class of a cloud storage, a secondperformance storage class of the cloud storage, or a third performancestorage class of the cloud storage.

At 212, a batch of data chunks is generated. The data chunks included inthe batch correspond to the one or more data chunks identified by thefile system manager of the cloud server.

At 214, it is determined whether a batch size of the generated batch isgreater than or equal to a threshold size and less than a maximum sizefor a cloud storage element object. The size of the generated batchvaries based on the number of data chunks included in the receivedportion of specified content that are already stored in cloud storage.

In the event the generated batch size is greater than or equal to thethreshold size (e.g., 4 MB) and less than a maximum size for a cloudstorage element object, process 200 proceeds to step 220. In the eventthe batch size of a batch is not greater than or equal to the thresholdsize, process 200 proceeds to 216. In the event the generated batch sizeis greater than the maximum size for a cloud storage element object, thegenerated batch is split into two or more batches where each batch isgreater than or equal to the threshold size and less than the maximumsize for a cloud storage element object. In such a scenario, process 200proceeds to 220 for each of the batches.

In some embodiments, at 214, it is determined whether the generatedbatch size is within a default size range for a cloud storage elementobject (e.g., 4 MB - 12 MB). In the event the size of the generatedbatch is within the default size range, process 200 proceeds to 220. Inthe event the generated batch size is not within the default size range,process 200 proceeds to 216.

At 216, it is determined whether a batch period is greater than or equalto a batch threshold period. For a generated batch that is less than athreshold size, data chunks included in the generated batch may bestored in memory for a batch threshold period (e.g., <1 second) to allowadditional data chunks to be included in the batch. Storing a batch ofdata chunks that is less than the threshold size in a cloud storageelement object at cloud storage may cause the recovery process of anobject associated with the cloud storage element object to be slowbecause additional reads need to be performed when restoring the objectassociated with the cloud storage element object.

In the event the batch period is greater than or equal to the batchthreshold period, process 200 proceeds to 218. In the event the batchperiod is not greater than or equal to the batch threshold period,process 200 returns to 204 where the client-side component processes asubsequent portion of the specified content to identify one or more datachunks of the subsequent portion of the specified content that are to bestored in the cloud storage. The one or more data chunks of thesubsequent portion are combined with the one or more data chunks of theprevious portion to generate a new batch of data chunks. The client-sidecomponent determines whether a size of the new batch of data chunks isgreater than the threshold size. If it is, then process 200 proceeds tostep 220. If it is not, then process 200 proceeds to step 216 and theabove process repeats until the batch threshold period is greater thanor equal to the batch threshold period.

At 218, the one or more data chunks included in a batch and an index ofthe one or more data chunks included in the batch are written to astorage of a cloud server included in the data plane. The one or moredata chunks may be provided to cloud server of the data plane via anencrypted channel, such as a gRPC connection. In some embodiments, theone or more data chunks are encrypted.

At 220, the data chunks included in the batch are encrypted using theencryption key received at 210. In some embodiments, the data chunks arecompressed prior to being encrypted. At 222, the encrypted data chunksare written to referenced portion of the cloud storage identified at210. In some embodiments, a cloud storage element object is generatedwith the one or more data chunks and written to the cloud storageidentified at 210. A cloud storage may be selected among a plurality ofdifferent cloud storage destinations associated with differentperformance storage classes. The selected cloud storage may be based ona service level agreement associated with a storage tenant.

At 224, an indication that the one or more identified data chunks arestored at the referenced portion of the cloud storage is provided to acloud server. The referenced portion of the cloud storage may includeone or more existing cloud storage element objects and/or one or morenew cloud storage element objects. A cloud server hosting a file systemmanager in the data plane, such as cloud server 113, may receive theindication. The indication may include the corresponding chunkidentifiers for the one or more data chunks that were sent at step 208as well as cloud storage locations for the one or more data chunks thatwere written to cloud storage at step 222. In response to receiving theindication, the file system manager is configured to generate metadata(e.g., tree data structure, chunk metadata data structure, cloud storageelement object metadata data structure) that enables the one or moredata chunks that were written to cloud storage to be located andprovides a view (partial or complete) of the source system at aparticular moment in time.

At 226, it is determined whether any additional portions of thespecified content have been received at the client-side component. Inthe event any additional portions of the specified content have beenreceived at the client-side component, process 200 returns to step 206.In the event any additional portions of the specified content have notbeen received at the client-side component, process 200 ends.

FIG. 3 is a flow diagram illustrating a process for determining whichdata chunks to provide to a cloud storage in accordance with someembodiments. In the example shown, process 300 may be implemented by afile system manager, such as file system manager 115.

At 302, data chunk metadata that includes one or more chunk identifiersassociated with one or more data chunks are received. A chunk identifiermay be a unique identifier, such as a SHA-1 identifier. The one or morechunk identifiers may be associated with a particular amount of data(e.g., 4 MB of data). The data chunk metadata may include otherinformation, such as data chunk size and object offset for a data chunk.The data chunk metadata indicates a sequence of data chunks within anobject.

At 304, a received chunk identifier is compared to identifiers includedin a deduplication table. The deduplication table may associate datachunks with their corresponding chunk identifiers. The deduplicationtable may identify some or all of the data chunks that are stored in acloud storage. In some embodiments, the one or more chunk identifiersnot included in the deduplication table correspond to data chunks thatare not stored at a cloud storage. In some embodiments, the one or morechunk identifiers not included in the deduplication table correspond toone or more data chunks that are stored at the cloud storage, but theone or more chunk identifiers associated with the one or more datachunks that are stored at cloud storage are not included indeduplication table due to one or more factors, such as whether the datachunk is referenced by a tree data structure that was generated after aparticular date, a recency of when the data chunk was last deduplicated,a frequency at which the data chunk is deduplicated, a service levelagreement, a storage tier of a data chunk, etc.

The cloud storage may be associated with a plurality of performancestorage classes. The deduplication table may indicate in whichperformance storage class of the cloud storage that a data chunk isstored. In some embodiments, the deduplication table only identifies thedata chunks that are stored in a top performance storage class of thecloud storage. In some embodiments, the deduplication table identifiesthe data chunks that are stored in the cloud storage based on a servicelevel agreement associated with a storage tenant. For example, theservice level agreement associated with a storage tenant may allow thestorage tenant to store data chunks in a first performance storage classand a second performance storage class. The deduplication table mayidentify the data chunks that are stored in the first performancestorage class of the cloud storage and the second performance storageclass of the cloud storage.

At 306, it is determined whether there is a match between the receivedchunk identifier and one of the chunk identifiers included in thededuplication table. In the event there is a match between the receivedchunk identifier and one of the chunk identifiers included in thededuplication table, process 300 proceeds to 310. In the event there isnot a match between the received chunk identifier and one of the chunkidentifiers included in the deduplication table, process 300 proceeds to308.

At 308, the received chunk identifier is included in a data structure.In some embodiments, the data structure indicates one or more chunkidentifiers associated with one or more data chunks that are not storedat a cloud storage. In some embodiments, the data structure indicatesone or more chunk identifiers associated with one or more data chunksthat are stored at the cloud storage, but the one or more data chunksthat are stored at the cloud storage are excluded from the deduplicationtable due to one or more factors, such as whether the data chunk isreferenced by a tree data structure that was generated after aparticular date, a recency of when the data chunk was last deduplicated,a frequency at which the data chunk is deduplicated, a service levelagreement, a storage tier of a data chunk, etc.

At 310, the received chunk identifier is excluded from the datastructure (e.g., list, table, etc.).

At 312, it is determined whether there are any more chunk identifiers tocompare to chunk identifiers included in the deduplication table. In theevent there are more chunk identifiers to compare to chunk identifiersincluded in the deduplication table, process 300 returns to 304. In theevent there are no more chunk identifiers to compare to chunkidentifiers included in the deduplication table, process 300 proceeds to314.

At 314, a data structure that includes one or more chunk identifiers isprovided to a client-side component. The data structure may be providedto a client-side component at step 208 of process 200.

FIG. 4 is a flow diagram illustrating a process for writing data chunksto cloud storage in accordance with some embodiments. In the exampleshown, process 400 may be performed by one or more cloud servers, suchas the one or more cloud servers 113.

At 402, a batch of one or more data chunks is received. A client-sidecomponent may have generated a batch of one or more data chunks, butdetermined that a batch size of the generated batch is less than athreshold size.

At 404, a plurality of data chunks from a plurality of batches areaggregated. The batch of one or more data chunks received at 402 may beaggregated with one or more other batches of data chunks. In someembodiments, the one or more other batches of data chunks are receivedfrom the same client-side component that provided the batch of one ormore data chunks received at 402. In some embodiments, the one or moreother batches of data chunks are received from a different client-sidecomponent associated with a tenant. The different client-side componentmay be located at the same datacenter (e.g., a second client-sidecomponent at 104) or located at a different datacenter (e.g., aclient-side component located at a datacenter different from datacenter102).

At 406, it is determined whether the cumulative size of the aggregateddata chunks is greater than a threshold size and less than a maximumsize for a cloud storage element object. In the event the cumulativesize of the aggregated data chunks is greater than or equal to athreshold size and less than the maximum size, process 400 proceeds to408. In the event the cumulative size of the aggregated data chunks isnot greater than or equal a cloud storage element size threshold,process 400 proceeds to 410.

At 408, the aggregated data chunks are encrypted, the encrypted datachunks are written to a cloud storage element object and stored at thecloud storage. In some embodiments, the data chunks are compressed priorto being encrypted.

At 410, it is determined whether a total time since receiving the batchof one or more data chunks at 402 is greater than a threshold time(e.g., 2 hours). In the event the total time since receiving the one ormore data chunks at 402 is greater than the threshold time, process 400proceeds to 408 where the aggregated data chunks are encrypted andwritten to a cloud storage element object stored at the cloud storageeven though the cumulative size of the aggregated data chunks is lessthan the threshold size. In some embodiments, the data chunks arecompressed prior to being encrypted.

In some embodiments, the cloud storage element object associated withthe cloud storage is a new cloud storage element object. In someembodiments, the cloud storage element object associated with the cloudstorage is a cloud storage element object yet to be created. In someembodiments, the cloud storage element object associated with the cloudstorage is an existing cloud storage element object. In the event thecloud storage element object associated with the cloud storage is anexisting cloud storage element object, the file system manager may readthe data chunks associated with the existing cloud storage elementobject, combine the read data chunks with one or more new data chunks togenerate a new version of the cloud storage element object, create anindex for the new version of the cloud storage element object, andinclude the index in the new version of the cloud storage elementobject. The index may include an object id and a version number.

A storage of a cloud server (e.g., EC2 instance) may be less reliableand/or more expensive than a cloud storage (e.g., S3). The one or moredata chunks may be written to a cloud storage element object stored atthe cloud storage to ensure that the one or more data chunks are notlost in the event the storage of the cloud server fails. In the eventthe total time since receiving the one or more data chunks at 402 is notgreater than the threshold time, process 400 returns to 404.

FIG. 5 is a flow diagram illustrating a process for generating metadatain accordance with some embodiments. In the example shown, process 500may be implemented by a cloud server, such as one of the one or morecloud servers 113.

At 502, an indication that a client-side component stored one or moredata chunks at a portion of a cloud storage is received from aclient-side component. The received indication may indicate theclient-side component has finalized sending the one or more data chunks.The received indication may be the indication provided at step 224 ofprocess 200 by a client-side component. The indication may include thecorresponding chunk identifiers for the one or more data chunks thatwere sent from client-side component at step 208 as well as cloudstorage locations for the one or more data chunks that were written tocloud storage at step 222.

At 504, metadata is generated for the one or more data chunks stored inthe cloud storage by the client-side component. The one or more datachunks stored in the cloud storage correspond to the content of a sourcesystem at a particular moment in time. The one or more data chunks maybe stored in one or more cloud storage element objects of the cloudstorage. In response to receiving the indication, the file systemmanager of the cloud server is configured to generate metadata (e.g.,tree data structure, chunk metadata data structure, cloud storageelement object data structure) that enables the one or more data chunksthat were written to cloud storage to be located and provides a view(partial or complete) of the source system at a particular moment intime.

Metadata may be generated that enables the data chunks that correspondto the content of the source system at the particular moment in time tobe located. The metadata may include one or more tree data structuresthat organize the data chunks stored in the one or more cloud storageelement objects. An example of the tree data structure is a snapshottree, which may be based on a B+ tree structure (or other type of treestructure in other embodiments). An example of a tree data structure isdescribed in U.S. Pat. Application 16/287,214 entitled “Deploying ACloud Instance Of A User Virtual Machine,” filed on Feb. 27, 2019.

A snapshot tree includes a root node, one or more levels of one or moreintermediate nodes associated with the root node, and one or more leafnodes associated with an intermediate node of the lowest intermediatelevel. The root node of a snapshot tree includes one or more pointers toone or more intermediate nodes. Each intermediate node includes one ormore pointers to other nodes (e.g., a lower intermediate node or a leafnode). A leaf node may store file system metadata (e.g., owner, created,last update, size, permissions, etc.), data associated with a contentfile that is less than or equal to a limit size (e.g., 256 kB), anidentifier of a data brick, one or more pointers to one or more metadatastructures (e.g., Blob structure), etc. A leaf node of the snapshot treemay correspond to an inode.

A metadata structure may be generated for an object (e.g., a contentfile, a virtual machine, a container, an application, a database, etc.)that is greater than the limit size (e.g., 256 kB) and was included inthe source system content that was backed up to the cloud storage. Themetadata structure is configured to store the metadata associated withan object that enables the data chunks associated with the object to belocated. The metadata structure includes a root node, one or more levelsof one or more intermediate nodes associated with the root node, and oneor more leaf nodes associated with an intermediate node of the lowestintermediate level. A metadata structure is similar to a snapshot tree,but a leaf node of a metadata structure includes an identifier of a databrick associated with one or more data chunks of the content file andmetadata associated with the one or more data chunks (e.g., chunkidentifier, cloud storage element object identifier, etc.). A leaf nodeof the snapshot tree may include a pointer to a root node of themetadata structure corresponding to an object.

The location of the one or more data chunks associated with a data brickmay be identified using one or more data structures (e.g., list, table,etc.). A first data structure (e.g., chunk metadata data structure) maystore information that associates a plurality of chunk identifiers withtheir corresponding cloud storage element object identifiers. Thisindicates that a data chunk having a particular chunk identifier isstored in a cloud storage element object having a particular cloudstorage element object identifier. In some embodiments, the chunkmetadata data structure is deduplication table 114. A second datastructure (e.g., cloud storage element object metadata data structure)may associate a cloud storage element object identifier associated witha cloud storage element object with one or more data chunks stored inthe cloud storage element object. The second data structure may alsoindicate a corresponding cloud storage element object offset for each ofthe one or more data chunks stored in the cloud storage element object.In some embodiments, the first data structure and the second datastructure are combined as a single data structure. The one or more treedata structures, the first data structure, and the second data structuremay be stored in metadata store 117, which may be stored in a memory ofcloud server 113.

The one or more data chunks associated with a data brick may be locatedbased on the chunk metadata data structure (e.g., a table) and the chunkstorage element object metadata data structure. For example, a firstdata brick having a first brick identifier may be associated with afirst chunk identifier (e.g., SHA-1 hash value). The first chunkidentifier may be used in conjunction with the chunk metadata datastructure to identify a cloud storage element object identifier. A cloudstorage element object having the identified cloud storage elementobject identifier is comprised of a plurality of data chunks. The cloudstorage element object metadata data structure may be used to identify acorresponding location of the plurality of data chunks. The cloudstorage element object metadata data structure may include correspondingoffset information of the plurality of data chunks within a cloudstorage element object.

At 506, the metadata for the one or more data chunks is serialized intoa data file comprising a flat set of data. The flat set of data includesa plurality of data blocks where each data block of the flat set of datacorresponds to a node of the tree data structure. A block thatcorresponds to a root node or intermediate node of the tree datastructure includes a file offset to another data block of a flat set ofdata. A data block that corresponds to a leaf node includes a referenceto a storage location for one or more data chunks with which the leafnode is associated. An example of a serialized data file is described inU.S. Pat. application 15/689,704 entitled “Snapshot Archive Management,”filed on Aug. 29, 2017, which is incorporated herein by reference forall purposes. The metadata may be stored in a metadata store, such asmetadata store 117.

At 508, the serialized metadata is stored in the cloud storage. Astorage of a cloud server (e.g., EC2 instance) that stores the metadatamay be less reliable and/or more expensive than the cloud storage (e.g.,S3). The metadata for the one or more data chunks may be serialized andstored at the cloud storage to ensure that the metadata for the one ormore data chunks is not lost in the event the storage of the cloudserver fails.

Steps 506 and 508 may be periodically (e.g., daily, weekly, bi-monthly,monthly, etc.) performed by a cloud server. In some embodiments, steps506 and 508 are performed after a particular amount of time has passedafter a backup has completed.

FIG. 6 is a flow diagram illustrating a process for managing adeduplication table in accordance with some embodiments. In the exampleshown, process 600 may be implemented by a file system manager, such asfile system manager 115. The deduplication table may be stored in asolid-state drive (SSD) of a cloud server. The amount of storage spacein the SSD is finite. A size of the deduplication table may grow as anamount of data stored by a storage tenant at a cloud storage increases.The deduplication table may be managed in a manner described by process600 to prevent the deduplication table from using an unnecessary amountof the SSD storage because the deduplication table includes referencesto data chunks that do not need to be deduplicated for one or moredifferent reasons. Process 600 may be performed for each of the chunkidentifiers included in a deduplication table. In some embodiments,process 600 is performed as a background process.

At 602, it is determined to reduce a storage size of a deduplicationtable. In some embodiments, the storage size of a deduplication tablehas exceeded a threshold max size. In some embodiments, a storage sizeof the deduplication table is periodically scanned (e.g., daily, weekly,monthly, etc.) to reduce the storage size of the deduplication table. Adeduplication table includes a plurality of entries. Each of the entriesassociates a corresponding chunk identifier to a specific data chunkstored at a cloud storage.

At 604, a specific entry is identified to be removed from thededuplication table. The specific entry may be identified based on oneor more properties of the specific entry or a specific data chunkcorresponding to the specific entry. For example, a specific entry maynot have been accessed within a threshold period of time. The specificdata chunk corresponding to the specific entry may be identified basedon whether the data chunk is referenced by a tree data structure thatwas generated after a particular date, a recency of when the data chunkwas last deduplicated, a frequency at which the data chunk isdeduplicated, etc.

In some embodiments, for the specific entry to be removed from thededuplication table, it is determined whether the data chunkcorresponding to the specific entry should be demoted to a lowerperformance storage class. A cloud storage may include a plurality ofperformance storage classes. The speed at which data may be accessed atthe cloud storage may depend on the performance storage class with whichthe data chunk is associated. In some embodiments, data chunks that areassociated with a top performance storage class are to be deduplicatedand include corresponding entries in the deduplication table. In someembodiments, data chunks that are not associated with the topperformance storage class are not to be deduplicated and not includecorresponding entries in the deduplication table.

A deduplicated data chunk may be determined to be demoted from the topperformance storage class to a lower performance storage class based onone or more factors. For example, the one or more factors may includebased on whether the data chunk is referenced by a tree data structurethat was generated after a particular date, a recency of when the datachunk was last deduplicated, a frequency at which the data chunk isdeduplicated, a service level agreement, a storage tier of a data chunk,etc.

In some embodiments, the one or more factors indicate that the datachunk should remain in its current performance storage class. In someembodiments, the one or more factors indicate that the data chunk shouldbe demoted to a lower performance storage class. In some embodiments,the one or more factors indicate that the data chunk should be upgradedto a higher performance storage class.

In the event it is determined that the data chunk should be demoted to alower performance storage class, the file system manager determineswhether one or more objects that reference the specific data chunk areassociated with a performance storage class that is lower than a currentperformance storage class associated with the specific data chunk. Forexample, the current performance storage class associated with thespecific data chunk may be performance storage class 123 and theperformance storage class associated with the one or more objects thatreference the specific data chunk may be performance storage class 124or performance storage class 125. In the event it is determined that thededuplicated data chunk should not be demoted to a lower performancestorage class, the deduplication table is maintained in its currentstate. In the event it is determined that the deduplicated data chunkshould be demoted to a lower performance storage class, the performancestorage class for the data chunk is modified to the lower performancestorage class and the entry corresponding to the data chunk is removedfrom the deduplication table.

At 606, the specific entry is removed from the deduplication tabledespite the specific data chunk corresponding to the specific entrystill being stored in the cloud storage. This may reduce the amount ofstorage used by the deduplication table in a SSD of a cloud server.

FIG. 7 is a flow diagram illustrating a process for updating a cloudstorage element object in accordance with some embodiments. In theexample shown, process 700 may be implemented by a cloud server, such asone of the one or more cloud servers 113.

At 702, a cloud storage is monitored. The cloud storage is configured tostore a plurality of cloud storage object elements.

At 704, a size of a cloud storage element object is determined. Forexample, the size of a cloud storage element object may be 0.5 MB whenthe default size of the cloud storage object element is 8 MB. In someembodiments, the cloud storage element object has an initial and currentsize that is less than the default size of a cloud storage objectelement or smaller than the default size range of a cloud storage objectelement. In some embodiments, the cloud storage element object has aninitial size that is the default size of a cloud storage object elementor within the default size range of a cloud storage object element and acurrent size of the cloud storage element object is less than thedefault size of a cloud storage object element or smaller than thedefault size range of a cloud storage object element.

At 706, it is determined whether the size of the cloud storage elementobject is less than a first threshold size. For example, the firstthreshold size may be 1 MB. In the event it is determined that the sizeof the cloud storage element object is less than the first thresholdsize, process 700 proceeds to 708. In the event it is determined thatthe size of the cloud storage element object is not less than the firstthreshold size, process 700 proceeds to 710.

At 708, the cloud storage element object is added to a set of one ormore cloud storage element objects available for update. An identifierof the cloud storage element object may be included in a set of one ormore cloud storage element object identifiers received by a client-sidecomponent at 210.

A client-side component may select the added cloud storage elementobject from the set of one or more cloud storage element objects andupdate the selected cloud storage element object with an updatedversion. The updated version may include previously existing data of thecloud storage element object and additional data. The previouslyexisting data and the additional data may be encrypted using anencryption key associated with a storage tenant.

In some embodiments, the updated version of the cloud storage elementobject associated with the cloud storage is a new cloud storage elementobject. For example, the cloud storage element object may be created bythe client-side component. In some embodiments, the updated version ofthe cloud storage element object associated with the cloud storage is anexisting cloud storage element object. In the event the updated versionof the cloud storage element object associated with the cloud storage isan existing cloud storage element object, the file system manager mayread the data chunks associated with the existing cloud storage elementobject, combine the read data chunks with one or more new data chunks togenerate a new version of the cloud storage element object, create anindex for the new version of the cloud storage element object, andinclude the index in the new version of the cloud storage elementobject. The index may include an object id and a version number.

At 710, it is determined whether the size of the cloud storage elementobject is less than a second threshold size. For example, the secondthreshold size may be 4 MB. In the event the size of the cloud storageelement object is less than a second threshold size, process 700proceeds to 712. In the event the size of the cloud storage elementobject is not less than the second threshold size, process 700 proceedsto 714.

At 712, new data chunks are stored in a new version of the cloud storageelement object. In some embodiments, the new data chunks are receivedfrom a client-side component. The cloud storage element object isupdated to an updated version that includes previously existing data ofthe cloud storage element object and the new data received from theclient-side component. The previously existing data and the new datareceived from the client-side component may be encrypted using anencryption key associated with a storage tenant and written to a newversion of the cloud storage element object.

In some embodiments, the new data chunks are received from a cloudserver. The cloud storage element object is updated to an updatedversion that includes previously existing data of the cloud storageelement object and the new data chunks received from the cloud server.The previously existing data and the new data chunks received from thecloud server may be encrypted using an encryption key associated with astorage tenant and written to a new version of the cloud storage elementobject.

In some embodiments, the previous version of the cloud storage elementobject is deleted.

At 714, a current version of the cloud storage element object ismaintained.

FIG. 8 is a flow diagram illustrating a process for updating a cloudstorage element object in accordance with some embodiments. In theexample shown, process 800 may be implemented by a client-sidecomponent, such as one of the one or more client-side components 104. Insome embodiments, process 800 is implemented to perform some of step 214of process 200.

At 802, a cloud storage element is selected for update from a set of oneor more cloud storage element objects available for update. The set ofone or more cloud storage element objects available for update may bereceived from a file system manager of a cloud server.

At 804, it is determined whether data associated with the cloud storageelement object is stored in a cache of the client-side component. In theevent data associated with the cloud storage element object is stored ina cache of the client-side component, process 800 proceeds to 806. Inthe event data associated with the cloud storage element object is notstored in the cache of the client-side component, process 800 proceedsto 808.

At 806, an updated version of the cloud storage element object thatincludes the data associated with the cloud storage element objectstored in the cache and additional data is generated. The dataassociated with the cloud storage element object and the additional datamay be encrypted using the encryption key provided at 210. An index forthe new version of the cloud storage element object may be generated andincluded in the new version of the cloud storage element object. Theindex may include an object id and a version number.

At 808, the cloud storage element object is read from the cloud storage.The client-side component may temporarily store the read data in a cacheof the client-side component.

At 810, an updated version of the cloud storage element object thatincludes the data associated with the read cloud storage element objectand additional data is generated. The data associated with the cloudstorage element object and the additional data may be encrypted usingthe encryption key provided at 210. An index for the new version of thecloud storage element object may be generated and included in the newversion of the cloud storage element object. The index may include anobject id and a version number.

The invention can be implemented in numerous ways, including as aprocess; an apparatus; a system; a composition of matter; a computerprogram product embodied on a computer readable storage medium; and/or aprocessor, such as a processor configured to execute instructions storedon and/or provided by a memory coupled to the processor. In thisspecification, these implementations, or any other form that theinvention may take, may be referred to as techniques. In general, theorder of the steps of disclosed processes may be altered within thescope of the invention. Unless stated otherwise, a component such as aprocessor or a memory described as being configured to perform a taskmay be implemented as a general component that is temporarily configuredto perform the task at a given time or a specific component that ismanufactured to perform the task. As used herein, the term ‘processor’refers to one or more devices, circuits, and/or processing coresconfigured to process data, such as computer program instructions.

A detailed description of one or more embodiments of the invention isprovided along with accompanying figures that illustrate the principlesof the invention. The invention is described in connection with suchembodiments, but the invention is not limited to any embodiment. Thescope of the invention is limited only by the claims and the inventionencompasses numerous alternatives, modifications and equivalents.Numerous specific details are set forth in the description in order toprovide a thorough understanding of the invention. These details areprovided for the purpose of example and the invention may be practicedaccording to the claims without some or all of these specific details.For the purpose of clarity, technical material that is known in thetechnical fields related to the invention has not been described indetail so that the invention is not unnecessarily obscured.

Although the foregoing embodiments have been described in some detailfor purposes of clarity of understanding, the invention is not limitedto the details provided. There are many alternative ways of implementingthe invention. The disclosed embodiments are illustrative and notrestrictive.

What is claimed is:
 1. A method, comprising: determining at a cloudserver that a first size of a first cloud storage element object is atleast below a first threshold; in response to a first determination,requesting a client-side component to store additional data in the firstcloud storage element object including by having the client-sidecomponent update the first cloud storage element object with an updatedversion that includes previously existing data of the first cloudstorage element object and the additional data; and adding the firstcloud storage element object to a set of one or more cloud storageelement objects available for update, wherein the client-side componentselects the first cloud storage element object from the set of one ormore cloud storage element objects available for update, wherein theclient-side component generates the updated version of the first cloudstorage element object that has a second size that is greater than orequal to the first threshold, wherein to generate the updated version ofthe first cloud storage element object, the client-side componentdetermines that the previously existing data of the first cloud storageelement object is not stored in a cache storage of the client-sidecomponent, and in response to determining that the previously existingdata of the first cloud storage element object is not stored in thecache storage of the client-side component, the client-side componentreads the first cloud storage element object, and uses the read data ofthe first cloud storage element object and additional data to generatethe updated version of the first cloud storage element object.
 2. Themethod of claim 1, further comprising monitoring a cloud storage storingthe first cloud storage element object.
 3. The method of claim 1,wherein an initial size of the first cloud storage element object wasabove the first threshold and a current size of the first cloud storageelement object is below the first threshold.
 4. The method of claim 1,wherein an initial size and a current size of the first cloud storageelement object are below the first threshold.
 5. The method of claim 1,wherein an identifier associated with the first cloud storage elementobject is included in the set of one or more cloud storage elementobjects available for update.
 6. The method of claim 1, wherein the readdata of the first cloud storage element object and the additional dataare encrypted using an encryption key received from the cloud server. 7.The method of claim 6, wherein the encryption key is for a particularstorage tenant.
 8. The method of claim 6, wherein the encryption keyenables read or write access for the first cloud storage element objectat a credential level.
 9. The method of claim 6, wherein the encryptionkey enables read or write access for the first cloud storage elementobject at a prefix level.
 10. The method of claim 9, wherein the prefixlevel is a partial file path in cloud storage.
 11. The method of claim9, wherein the prefix level is a full file path in cloud storage. 12.The method of claim 6, wherein the encryption key expires after aparticular amount of time.
 13. The method of claim 1, wherein theupdated version of the first cloud storage element object is associatedwith a corresponding identifier and corresponding version number. 14.The method of claim 1, further comprising: determining that a third sizeof a second cloud storage element object is above the first thresholdbut below a second threshold; and in response to a second determination,storing new data chunks in the second cloud storage element objectincluding by updating the second cloud storage element object to be anupdated version of the second cloud storage element object that includespreviously existing data of the second cloud storage element object andthe new data chunks.
 15. The method of claim 14, wherein the client-sidecomponent encrypts the previously existing data of the second cloudstorage element object and the new data chunks using an encryption keyassociated with a storage tenant.
 16. A computer program productembodied in a non-transitory computer readable medium and comprisingcomputer instructions for: determining at a cloud server that a size ofa first cloud storage element object is at least below a firstthreshold; in response to a first determination, requesting aclient-side component to store additional data in the first cloudstorage element object including by having the client-side componentupdate the first cloud storage element object with an updated versionthat includes previously existing data of the first cloud storageelement object and the additional data; and adding the first cloudstorage element object to a set of one or more cloud storage elementobjects available for update, wherein the client-side component selectsthe first cloud storage element object from the set of one or more cloudstorage element objects available for update, wherein the client-sidecomponent generates the updated version of the first cloud storageelement object that has a second size that is greater than or equal tothe first threshold, wherein to generate the updated version of thefirst cloud storage element object, the client-side component determinesthat the previously existing data of the first cloud storage elementobject is not stored in a cache storage of the client-side component andin response to determining that the previously existing data of thefirst cloud storage element object is not stored in the cache storage ofthe client-side component, the client-side component uses the previouslyexisting data of the first cloud storage element object stored in thecache storage of the client-side component and additional data togenerate the updated version of the first cloud storage element object.17. The computer program product of claim 16, further comprisingcomputer instructions for monitoring a cloud storage storing the firstcloud storage element object.
 18. The computer program product of claim16, wherein an initial size of the first cloud storage element objectwas above the first threshold and a current size of the first cloudstorage element object is below the first threshold.
 19. The computerprogram product of claim 16, wherein an initial size and a current sizeof the first cloud storage element object are below the first threshold.20. A system, comprising: one or more processors configured to:determine at a cloud server that a first size of a first cloud storageelement object is at least below a first threshold; in response to afirst determination, request a client-side component to store additionaldata in the first cloud storage element object including by having theclient-side component update the first cloud storage element object withan updated version that includes previously existing data of the firstcloud storage element object and the additional data; and add the firstcloud storage element object to a set of one or more cloud storageelement objects available for update, wherein the client-side componentselects the first cloud storage element object from the set of one ormore cloud storage element objects available for update, wherein theclient-side component is configured to generate the updated version ofthe first cloud storage element object that has a second size that isgreater than or equal to the first threshold, wherein to generate theupdated version of the first cloud storage element object, theclient-side component determines that the previously existing data ofthe first cloud storage element object is not stored in a cache storageof the client-side component and in response to determining that thepreviously existing data of the first cloud storage element object isnot stored in the cache storage of the client-side component, theclient-side component uses the previously existing data of the firstcloud storage element object stored in the cache storage of theclient-side component and additional data to generate the updatedversion of the first cloud storage element object; and a memory coupledto the one or more processors and configured to provide the one or moreprocessors with instructions.